Resolves conflict in clickhouse-datalake{,-partitioned}: upstream switched
the datalake variants from filesystem-cache to userspace page-cache (PR #818).
The refactored install/query scripts now adopt the page-cache approach.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

mongodb: query takes a MongoDB aggregation pipeline (Extended JSON, one
line) on stdin instead of SQL — these are the same canonical 43 ClickBench
queries, just expressed as mongo pipelines. queries.txt is generated from
queries.js (the source of truth) by replacing JS-only constructors
(NumberLong, ISODate, NumberDecimal) with their EJSON canonical form. The
shim sets BENCH_QUERIES_FILE=queries.txt to point the driver at it.

polars: wrapped in a FastAPI server analogous to polars-dataframe, but the
load step uses pl.scan_parquet (LazyFrame) so the parquet file remains
needed at query time — the load script does NOT delete hits.parquet.
data-size returns the on-disk parquet size since a LazyFrame has no
materialized in-memory size.

Both systems now expose the standard install/start/check/stop/load/query/
data-size scripts and a 4-line benchmark.sh shim, removing the old
benchmark.sh / run.js / query.py / formatResult.js paths.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

… readiness

Per review (alexey-milovidov): clickhouse start leaves the system in the
desired state (server running) even when it returns non-zero with "already
running". Make the shared driver tolerate non-zero from ./start and rely on
bench_check_loop as the authoritative readiness signal. This lets per-system
start scripts stay simple — they just need to make a best-effort attempt to
launch.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…-system layout

These four entries were added on main while this branch was in flight (the
existing trino/ scripts here were a memory-connector stub that never worked
end-to-end). Rebuild each one against the new install/start/check/stop/load/
query/data-size contract so they share lib/benchmark-common.sh:

- trino, trino-partitioned: Hive connector + file metastore + local Parquet
  hardlinked into data/hits/ (matches main's working impl from PR #856).
- trino-datalake{,-partitioned}: same, plus the AnonymousAWSCredentials shim
  to read clickhouse-public-datasets/hits_compatible/athena from anonymous
  S3 (the published bucket size is reported by data-size since the data is
  read on demand). BENCH_DOWNLOAD_SCRIPT="" — no local dataset to fetch.
- benchmark.sh in all four becomes a 4-line shim. Old run.sh deleted.

…r-system layout

These four entries were added on main while this branch was in flight.
Adapt them to the install/start/check/stop/load/query/data-size contract:

- presto, presto-partitioned: Hive connector + file metastore + local Parquet
  hardlinked into data/hits/.
- presto-datalake{,-partitioned}: same plus the AnonymousAWSCredentials shim
  (compiled in a throwaway trinodb/trino container, since the prestodb image
  ships only a JRE) so the hive-hadoop2 plugin can read the public bucket
  anonymously. BENCH_DOWNLOAD_SCRIPT="" — schema-only load against S3.

Each benchmark.sh becomes a 4-line shim. Old run.sh deleted.

These two entries were added on main while this branch was in flight.
Adapt to the install/start/check/stop/load/query/data-size contract:

- BENCH_DOWNLOAD_SCRIPT="" — the vortex bench binary fetches Parquet and
  converts to .vortex on first invocation.
- BENCH_RESTARTABLE=no — embedded Rust CLI; per-query restart would
  dominate query time.
- query: stages stdin into a temp queries-file and passes -q 0, since the
  bench binary addresses queries by index rather than reading SQL on stdin.
- The single variant uses the `clickbench` binary (vortex 0.34.0); the
  partitioned variant uses `query_bench clickbench` (vortex 0.44.0). Old
  run.sh deleted.

Quickwit was added on main while this branch was in flight. Adapt to the
install/start/check/stop/load/query/data-size contract:

- BENCH_QUERIES_FILE="queries.json" — Quickwit accepts Elasticsearch-format
  JSON queries via the /_elastic compat API, not SQL. queries.json holds one
  ES query per line; queries not expressible in Quickwit are encoded as the
  literal "null".
- BENCH_DOWNLOAD_SCRIPT="" — the load script fetches hits.json.gz directly
  (there is no shared download-hits-json helper) and pipes it through
  `quickwit tool local-ingest`, since v0.9's sharded ingest-v2 endpoint caps
  single-node throughput at a few MB/s.
- BENCH_RESTARTABLE=yes — relies on the common driver's per-query restart
  to flush Quickwit's fast_field_cache and split_footer_cache (the result
  caches are already disabled in node-config.yaml).
- query: returns non-zero for "null" queries so the framework records null
  in the per-query timing array; otherwise reports .took (ms → seconds).

Old run.sh deleted.

The original used /tmp/gizmosql_server_$$.pid where $$ is the calling
process's PID. That worked when benchmark.sh sourced util.sh and called
start/stop in the same shell, but under the new per-system layout each of
start, stop, load, and query sources util.sh in its own subshell — so
stop_gizmosql couldn't find the PID file written by start_gizmosql. Use a
fixed path under the system directory instead. Also expose wait_for_gizmosql
so callers (like load) can wait for readiness without restarting.

Conflict only in gizmosql/benchmark.sh — kept the thin shim. Main switched
gizmosql to the official one-line installer (PR #879); fold that into
gizmosql/install so we stop hand-detecting arch and downloading the zip.

Other changes auto-merged: quickwit/index_config.yaml gained tag_fields on
CounterID + record:basic on text fields (PR #886), and assorted result
JSONs for ClickHouse Cloud / Citus / Cratedb / etc.

start/stop scripts may emit progress lines (clickhouse-server prints PID
table tracking, sudo's chown invocation, postgres's startup messages,
etc.). With BENCH_RESTARTABLE=yes those scripts run before every query,
so their output interleaves with the parseable [t1,t2,t3] / Load time /
Data size lines and breaks the cloud-init log POST to play.clickhouse.com.

Redirect both stdout and stderr from ./start and ./stop to /dev/null at
the three call sites in lib/benchmark-common.sh. The check loop is the
authoritative readiness signal, so losing start's output costs nothing
in steady state; for debugging, run ./start manually outside the driver.

The DuckDB installer at install.duckdb.org drops the binary into
~/.duckdb/cli/latest/duckdb and only suggests adding that directory to
PATH. Previously each install attempted a per-user symlink into
~/.local/bin, which silently no-ops when that directory isn't on PATH
(default for root in cloud-init). The result was ./check failing for
300s with no useful error.

Symlink to /usr/local/bin/duckdb via sudo right after install instead;
that's on PATH for every user, and the symlink is itself idempotent.

Ubuntu's docker.io ships the docker CLI without the v2 compose plugin, so
the existing `command -v docker` short-circuit skipped installation on
boxes that already had docker but no `docker compose`. ./start then ran
`docker compose up -d`, which silently failed, and ./check timed out at
300s. Fall back to docker-compose-v2 for the Ubuntu package name.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Throughput variant of ClickBench. N connections (default 10) hold open
sessions and each picks a uniformly random query from the standard
43-query set; the run goes for a fixed wall-clock window (default 600s)
after a warmup. Reports completed queries, QPS, latency p50/p95/p99,
and per-query mean.

Backends: ClickHouse over HTTP (stdlib http.client), StarRocks over the
MySQL wire protocol (pymysql). Each system's recommended path so neither
is paying a wire-format penalty the other isn't.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…ned}/query: pass query via temp file

`python3 - <<'PY' ... PY` directs the heredoc into python3's stdin so the
interpreter can read its program from there. Once the heredoc is fully
consumed, sys.stdin (the same FD) is at EOF — so sys.stdin.read() inside
the heredoc returned an empty string, and chdb / hyper / sail dutifully
ran the empty query and reported ~0.000s for every try.

Stage stdin into a temp file in bash before invoking the heredoc and pass
the path as argv[1]; the python script reads the query from that file.

Also include result materialization in the timing window for chdb/query
and chdb-parquet-partitioned/query (move `end = ...` past fetchall /
str(res)) — the timer was previously stopped before the result was
realized, which would have under-counted query time even when the stdin
bug wasn't masking it entirely.

Right now ./check stderr is silently dropped while the loop retries for
300s, then we report "did not succeed within 300s" with no clue why.
For deterministic failures (missing env var like YT_PROXY for chyt, an
install step that didn't run, etc.) the user wastes 5 minutes and still
has to dig through the per-system check script to find out what
happened. Capture the last attempt's stderr and print it on timeout.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

The upstream install path assumes RHEL/Rocky/Alma — yum, grubby, SELinux,
the wheel group, /data0. On Ubuntu/Debian the prereqs phase silently
half-completes (several |||| true skips), the gpadmin user is sometimes
not created, and db-install would later die at `yum install -y go`.
Either way ./check times out at 300s with no diagnostic. Bail with a
clear "needs yum" message before doing anything destructive, and call
out the requirement in the README.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>